Visual Manipulation Relationship Network

Robotic grasping is one of the most important fields in robotics and convolutional neural network (CNN) has made great progress in detecting robotic grasps. However, including multiple objects in one scene can invalidate the existing grasping detection algorithms based on CNN because of lacking of manipulation relationships among objects to guide the robot to grasp things in the right order. Therefore, the manipulation relationships are needed to help robot better grasp and manipulate objects. This paper presents a new CNN architecture called Visual Manipulation Relationship Network (VMRN) to help robot detect targets and predict the manipulation relationships in real time. To implement end-to-end training and meet real-time requirements in robot tasks, we propose the Object Pairing Pooling Layer (OPL) to help to predict all manipulation relationships in one forward process. In order to train VMRN, we collect a dataset named Visual Manipulation Relationship Dataset (VMRD) consisting of 5185 images with more than 17000 object instances and the manipulation relationships between all possible pairs of objects in every image, which is labeled by the manipulation relationship tree. The experiment results show that the new network architecture can detect objects and predict manipulation relationships simultaneously and meet the real-time requirements in robot tasks.